It is known that operators of video-based missile targeting or imaging systems require the assistance of an automatic target cuer for quickly locating moving targets. This task is especially difficult when the targeting device is also traveling at a high rate of speed. As such, video-based missile targeting or imaging systems require a steady image sequence, which is only obtainable from a reliable platform. Typically, these platforms, such as ships, planes, armored vehicles and the like, require the use of inertial stabilized gimbaling systems which tend to be bulky and expensive. A steady image sequence is important when one considers that an object to be tracked, such as an adversarial missile, makes up just one pixel out of a 256.times.256 pixel array and is very difficult to observe. This difficulty in object tracking is especially true when a missile blends in with the background scene motion.
Current target cuer-systems or object tracking systems analyze each video image display, frame by frame. All objects within each frame are segmented based on their contrast level. In other words, characteristics such as brightness and density are compared between successive frames of video. Subsequently, each frame of video is validated, with the validated resolution processed by the tracking system. However, because these moving objects are ultimately tracked by comparing static frames of video, any movement at the pixel level is not considered by the tracking system. Therefore, current cuer systems have difficulty in tracking camouflaged targets, small targets, and targets that have a low signal to noise ratio because they lack the required contrast.
To overcome the difficulty of tracking targets that are camouflaged, small or have a low signal to noise ratio, motion detection enhancements can be added to the tracking system. Motion detection enhancements supplement conventional intensity based cuer systems by employing pixel motion data. In other words, if an object within a video display is moving at a rate of speed different than the remaining objects within a video display, the pixels assigned to that moving object are identifiable.
It will be appreciated that in a video-based system, a sensor, typically a high-speed camera, is moveable in a variety of directions, including but not limited to zoom, roll, pan, along-track translation, and cross translation. In addition to the aforementioned sensor movements, the pitch of the sensor causes non-uniform pixel motion, sometimes called keystoning, in the video display. Keystoning occurs when the outer portions of the video display move faster than the inner portions. As such, motion detection systems must compensate and remove these adverse keystoning effects and other undesired sensor movements. It has been found that the use of correlation techniques to compensate for this complex scene motion with the required accuracy to allow scene stabilization is not practical. Correlation techniques fail because only large groups of pixels are compared to one another. As such, motion of the individual pixels within the group of pixels is not detectable, especially in high clutter or camouflaged areas of view. However, it has been found that optical flow techniques can estimate individual pixel velocities with sufficient accuracy to allow for compensation when the sensor is viewing a scene containing complex scene motion.
Optical flow estimation allows estimation of background motion and irregular object motion so that undesired motion can be removed from the input image by "warping" the pixels on a sub-pixel level. In other words, the velocity movement or "optical flow" of each pixel is observed and processed at a sub-pixel level. This optical flow information is used to remove platform motion and to compare two video images in time to one another. All optical flow determinations require that a velocity continuity constraint equation be used.
Known methods of determining optical flow include employing a low pass filter to constrain the instantaneous velocity of each individual pixel, or employing a set of linear equations to fit the instantaneous pixel velocity data. Unfortunately, a low pass filter only considers spatially local pixel velocities and does not have the capability to consider the overall or global pixel velocity field. As such, tracking multiple objects by use of a low pass filter cannot respond to rapidly changing platform motion. Consequently, an optical flow system employing only a low pass filter does not have the desired sensitivity. Determining optical flow by employing linear equations to fit the instantaneous pixel velocity data does not properly consider sensor motion in the second degree. In other words, any curvilinear motion of the sensor is not considered in the optical flow determination. As such, the predicted motion represented by a linear equation includes inherent displacement errors because of the curvilinear motion of the background pixel motion.
Therefore, to overcome the aforementioned shortcomings in using an optical flow technique, there is a need in the art for an optical flow detection system which employs a more precise method of evaluating and filtering the instantaneous pixel velocity data. A device to meet this need is required to positively identify multiple threatening targets by using multiple frames of video imagery to estimate the velocity of each and every pixel image. Additionally, a device to meet this need is required to compensate for irregular platform motion.